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APPARATUS AND METHOD FOR IMPROVING VISIBILITY IN A MOTOR VEHICLE 



Background of the Invention 

The invention relates to an apparatus and a method for improving visibility in a motor vehicle 
having at least one infrared-sensitive image sensor system. 



5 The visibility conditions experienced by the driver of a motor vehicle influence the frequency 
of accidents in street traffic, and thus influence traffic safety. In addition to lighting 
conditions, weather conditions determine the driver's ability to see. For example, it is known 
that a higher percentage of fatal traffic accidents happen at night. Fog, rain, and snow also 
make visibility conditions worse. Especially poor conditions are present when the lighting 
10 conditions and the weather conditions are both poor, e.g. when rain reduces visibility at night. 



From DE 40 32 927 C2, an apparatus is known for improving visibility conditions in a motor 
vehicle having an infrared-sensitive camera and a display device formed as a head-up display. 
It is proposed that, as information for the driver, the camera image be superposed as a virtual 
15 image of the outer landscape. In order to illuminate the field of view acquired by the infrared- 
sensitive camera, a radiation source having an infrared radiation portion is used. The 
evaluation of the display is left to the driver. 



Advantages of the Invention 

20 The apparatus and the method of the present invention result in an improvement of the 
visibility in a motor vehicle while at the same time reducing stress on the driver. This 
advantage is achieved by controlling the at least one signaling means for producing the driver 
information, as a function of the course of the roadway. Drivers of today's modem motor 
vehicles have to process a large amount of information from street traffic and from the motor 

25 vehicle itself. The apparatus described below and the method enable an advantageous 

reduction in the amount of information that has to be processed by the driver. This results, 
particularly advantageously, in a high degree of acceptance by the driver. In addition, the 
apparatus and the method for reducing the number of accidents in street traffic in poor 
lighting and/or weather conditions can contribute in particular to a reduction in the number of 

30 accidents that take place at night. 
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Due to the use of spatially high-resolution image sensor systems, for example having a 
resolution of 1280x1024 pixels, the number of objects that can be acquired in the surrounding 
environment of the motor vehicle is high. The allocation of the acquired objects in the field of 
5 detection of the image sensor system to the course of the roadway provides a simple decision 
criterion for reducing the number of objects to be represented. By controlling the at least one 
signaling means for producing the driver information as a function of the position of at least 
one object in relation to the course of the roadway, a stimulus overload of the driver is 
advantageously prevented. It is especially advantageous that the apparatus and the method 
10 described below reduce the distraction of the driver to a minimum. This advantageously 

increases overall traffic safety. In addition, the allocation of the acquired objects to the course 
of the roadway as a decision criterion enables the use of simple, economical processing units. 

The controlling of the signaling means for producing the driver information as a function of 
15 the driving situation and/or the visibility conditions results in additional advantages of the 

apparatus described below and of the method. In this way, a further reduction of the quantity 
of information that a driver of a motor vehicle must process is advantageously achieved by 
displaying only information that is important for the driver. For example, the apparatus 
described below and the method enable a situation-specific warning of the driver concerning 
20 obstacles in the course of the roadway, i.e., in the driver's own lane and in adjacent areas, that 
the driver does not see when traveling at night with low beams. The controlling of the 
signaling means for producing the driver information as a function of the visibility conditions 
makes it possible to warn the driver so that he can react in timely fashion to an obstacle. In 
this way, accidents are advantageously prevented. However, at the same time it is possible to 
25 omit a warning of an obstacle if the obstacle, not visible to the driver, does not present a 
danger. 

Items of driver information, suitable for the representation of at least one object and/or of the 
course of the roadway, make it possible for the driver advantageously to register the course of 
30 the roadway and to rapidly and reliably himself recognize dangers presented by objects. For 
example, at night it is often difficult for the driver to see the course of the roadway itself. 
Through the simultaneous representation of the course of the roadway and of objects, a driver 
can judge whether objects represent a danger. 
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Through the use of at least one light pulse in the field of view of the driver in order to warn 
him, and/or at least one warning symbol, and/or at least one image marking, and/or at least 
one segment of an image, and/or at least one acoustic signal, and/or at least one optical signal 
as items of driver information, the quantity of information affecting the driver is reduced. 
5 Through this well-directed warning of the driver via one or more possible information paths, 
the danger of distracting the driver is advantageously reduced. 

By using at least one source of infrared radiation to illuminate at least a part of the 
surrounding environment of the motor vehicle, acquired by the infrared-sensitive image 
10 sensor system, the sensitivity and the spatial area of acquisition of the image sensor system 
are advantageously increased. The infrared radiation source makes it possible for the at least 
one image sensor system to acquire objects that do not give off infrared radiation. 

A computer program having program code means is especially advantageous for the 
15 execution of all steps of the method described below, if the program is executed on a 

computer. The use of a computer program enables the rapid and economical adaptation of the 
method to different image sensor systems and/or signaling means. 

Additional advantages result from the following description of exemplary embodiments, with 
20 reference to the Figures, and from the dependent patent claims. 

Drawing 

The present invention is explained in more detail below on the basis of the specific 
embodiments shown in the drawing. 
25 Figure 1 shows an overview drawing of a first exemplary embodiment for improving the 
visibility in a motor vehicle, 

Figure 2 shows an overview drawing of a second exemplary embodiment for improving the 
visibility in a motor vehicle, 

30 

Figure 3 shows a block diagram of the apparatus for improving the visibility in a motor 
vehicle, 

Figure 4 shows a flow diagram. 

35 
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Exemplary Embodiments 

In the following, an apparatus and a method for improving the visibility in a motor vehicle 
having an infrared-sensitive image sensor system are described. Via a signaling means, for 
example a monitor or a head-up display, the driver of the motor vehicle is given driver 
5 information concerning the course of the roadway and/or objects in the surrounding 

environment of the motor vehicle. The displaying of the driver information here takes place 
as a function of the course of the roadway. 

Figure 1 shows an overview drawing of a first exemplary embodiment for improving the 

10 visibility in a motor vehicle 10, made up of an infrared-sensitive image sensor system 12, a 
headlight 16, and an image on a display screen 26. Image sensor system 12 is situated in the 
interior of motor vehicle 10, behind the windshield, in the area of the interior rearview mirror. 
Image sensor system 12 is oriented in such a way that area of acquisition 14 of image sensor 
system 12 extends to the surrounding environment of motor vehicle 10 in the direction of 

15 travel. Image sensor system 12 includes an infrared-sensitive video camera having a CMOS 
image sensor and/or a CCD image sensor. The image sensor of image sensor system 12 
acquires at least near infrared radiation in the wavelength range between 780 nm and 1000 
nm. Motor vehicle 10 is driven by a driver, motor vehicle 10 being situated on a street and 
traveling in the direction of travel. In the situation shown in Figure 1, the weather conditions 

20 and the lighting conditions are poor, because darkness is adversely affecting the driver's view. 
Two headlights 16, situated at the right and at the left in the front area of motor vehicle 10, in 
the vicinity of the bumper, illuminate the surrounding environment of motor vehicle 10 in the 
direction of travel. In Figure 1, in a simplified representation only one headlight 16 is shown. 
In addition to the low-beam light 18 in the visible light spectral range, headlights 16 also 

25 produce radiation having a high-beam characteristic 20 in the infrared spectral range. The 

range of low-beam 18 is approximately 40 meters. Headlights 16 have a high-beam function 
in the visible spectral range, with which the driver can see up to 200 meters, depending on 
weather conditions. In this exemplary embodiment, the high-beam function in the visible 
spectral range is not activated. The radiated spectrum of the halogen lamps of headlights 16 

30 contain a large infrared portion that is radiated by these modified headlights 16 with a high- 
beam characteristic, in a manner invisible to human beings. The driver's perception 22 is 
strongly limited by the darkness. In contrast, infrared-sensitive image sensor system 12 
achieves a better perception 24 through the modification of front headlights 16, having 
radiation at least of near infrared radiation with wavelengths between 780 and 1000 nm with 
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high-beam characteristic 20. In the perception 24 of infrared-sensitive image sensor system 
12, an oncoming motor vehicle 28, a pedestrian 30, and roadway 32 can be seen, while in 
driver's perception 22 only headlights 29 of oncoming motor vehicle 28 can be recognized. In 
perception 24 of image sensor system 12, pedestrian 30, crossing behind oncoming motor 
5 vehicle 28, can be recognized clearly, while in the driver's perception 22 he is not visible. 
Perception 24 of infrared-sensitive image sensor system 12 is processed and is displayed, as 
an image on a display screen 26, to the driver of motor vehicle 10. As a display screen, a 
monitor is used that is situated in the dashboard of motor vehicle 10. This display screen can 
be situated in the area of the center console and/or in the multi-instrument panel of motor 

10 vehicle 10, immediately behind the steering wheel. Besides the displaying of the speed and/or 
RPM by the multi-instrument panel, it is also possible for the driver to see the image of the 
surrounding environment of motor vehicle 10 without changing the direction of his gaze. 
With the aid of suitable image processing algorithms, objects 28, 30 in the field of detection, 
i.e. in the area of the course of the roadway, are recognized, and objects 28, 30 are allocated s 

15 to the course of the roadway. Roadway 32 here fundamentally includes the driver's lane and 
the lane of the oncoming traffic. On highways, roadway 32 is formed by at least the driver's 
lanes. Roadway 32 is defined by roadway markings such as guideposts and/or lane marking 
lines. The course of the roadway here includes roadway 32 itself and areas adjacent to 
roadway 32, such as for example edge strips and/or footpaths and/or bicycle paths and/or 

20 entrances of streets. In this exemplary embodiment, an oncoming motor vehicle 28 and a 

pedestrian 30 are recognized as objects 28, 30. In the image on display screen 26, the image 
recorded by infrared-sensitive image sensor system 12 of the surrounding environment of 
motor vehicle 10 is displayed with a marking 34 of oncoming motor vehicle 28, a marking 36 
of pedestrian 30, and a marking of the course 38 of roadway 32. This image on display screen 

25 26, with inserted markings 34, 36, 38, is shown to the driver of motor vehicle 10. Through 
markings 34, 36, 38, the driver recognizes objects 28, 30 significantly faster than would be 
the case if only the unprocessed image information of infrared-sensitive image sensor system 
12 were displayed, and can easily allocate them to his driving lane. In this exemplary 
embodiment, the image processing algorithms are designed so that the course of the roadway 

30 38 is always marked, while markings 34, 36 of objects 28, 30 are executed only if objects 28, 
30 are situated in the area of the course of the roadway of motor vehicle 10. In an 
embodiment, the optical warning via markings 34, 36, 38 is additionally supported by an 
acoustic warning if a dangerous situation is recognized. Via a loudspeaker, the driver of 
motor vehicle 10 can be additionally warned acoustically. In this exemplary embodiment, a 
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dangerous situation is recognized because pedestrian 30 is crossing roadway 32 behind 
oncoming motor vehicle 28, and there is the danger of a collision of the driver's vehicle 10 
with pedestrian 30. 

5 Figure 2 shows an overview drawing of a second exemplary embodiment for improving 
visibility in a motor vehicle 10, made up of an infrared-sensitive image sensor system 12, a 
headlight 16, and the actual view of driver 40 of motor vehicle 10. As in the first exemplary 
embodiment according to Figure 1, image sensor system 12 is situated in the interior of motor 
vehicle 10, behind the windshield, in the area of the interior rearview mirror. Image sensor 

10 system 12 is oriented in such a way that area of acquisition 14 of image sensor system 12 

extends to the surrounding environment of motor vehicle 10 in the direction of travel. Image 
sensor system 1 2 includes an infrared-sensitive video camera having a CMOS image sensor 
and/or a CCD image sensor. The image sensor of image sensor system 12 acquires at least 
near infrared radiation in the wavelength range between 780 nm and 1000 nm. Motor vehicle 

15 1 0 is driven by a driver, motor vehicle 1 0 being situated on a street and traveling in the 

direction of travel. The same driving situation is present as in the first exemplary embodiment 
according to Figure 1 . The weather conditions and the lighting conditions are poor, because 
darkness is adversely affecting the driver's view. Two headlights 16, situated at the right and 
at the left in the front area of motor vehicle 10, in the vicinity of the bumper, illuminate the 

20 surrounding environment of motor vehicle 10 in the direction of travel. In Figure 2, in a 

simplified representation only one headlight 16 is shown. In addition to the low-beam light 
18 in the visible spectral range, headlights 16 also produce radiation having a high-beam 
characteristic 20 in the infrared spectral range. The range of low-beam light 18 is 
approximately 40 meters. Headlights 16 have a high-beam function in the visible spectral 

25 range, with which the driver can see up to 200 meters, depending on weather conditions. In 
this exemplary embodiment, the high-beam function in the visible spectral range is not 
activated. The radiated spectrum of the halogen lamps of headlights 16 contain a large 
infrared portion that is radiated by these modified headlights 16 with a high-beam 
characteristic, in a manner invisible to human beings. The driver's perception 22 is strongly 

30 limited by the darkness. In contrast, infrared-sensitive image sensor system 12 achieves a 

better perception 24 through the modification of front headlights 16, having radiation at least 
of near infrared radiation with wavelengths between 780 and 1000 nm with high-beam 
characteristic 20. In the perception 24 of infrared-sensitive image sensor system 12, an 
oncoming motor vehicle 28, a pedestrian 30, and roadway 32 can be seen, while in driver's 
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perception 22 only headlights 29 of oncoming motor vehicle 28 can be recognized. In 
perception 24 of image sensor system 12, pedestrian 30, crossing behind oncoming motor 
vehicle 28, can be seen clearly, while in driver's perception 22 he is not visible. Figure 2 
additionally shows the actual view of driver 40, including steering wheel 42, windshield 44, 
5 and dashboard 46. In this second exemplary embodiment, perception 24 of infrared-sensitive 
image sensor system 1 2 is supplied to a processing unit that produces a warning only if a 
dangerous situation is recognized. With the aid of suitable image processing algorithms, 
objects 28, 30 in the field of detection, i.e. in the area of the roadway, are recognized, and 
objects 28, 30 are allocated to the course of the roadway. As in the exemplary embodiment 

10 according to Figure 1, roadway 32 here fundamentally includes the driver's lane and the lane 
of the oncoming traffic. On highways, roadway 32 is formed by at least the driver's lanes. 
Roadway 32 is defined by roadway markings such as guideposts and/or lane marking lines. 
The course of the roadway here includes roadway 32 itself and areas adjacent to roadway 32, 
such as for example edge strips and/or footpaths and/or bicycle paths and/or entrances of 

15 streets. In this exemplary embodiment, an oncoming motor vehicle 28 and a pedestrian 30 are 
recognized as objects 28, 30. The processing unit recognizes the dangerousness of the 
situation. Via a simple projection device, in this exemplary embodiment a simple head-up 
display, on windshield 44 a small warning symbol 34, 36 is produced as marking 34 of 
oncoming motor vehicle 28 and as marking 36 of pedestrian 30 in the direction of view of the 

20 driver, at the position, thus determined, of oncoming motor vehicle 28 and of pedestrian 30. 
In this way, the driver is made to direct his gaze in the direction in which objects 28, 30 will 
later actually appear to the driver as obstacles in his field of view. In this exemplary 
embodiment, a colored marking in the form of a red and/or yellow triangle is used as warning 
symbol 34, 36. Alternatively, it is possible, using a short light impulse, to cause the driver to 

25 direct his gaze in the direction of the obstacle, using a projection device. The view of driver 
40 of the surrounding environment of motor vehicle 10 according to Figure 2 accordingly 
includes the light 48 from the headlights of oncoming motor vehicle 28, a marking 34 of 
oncoming motor vehicle 28, and a marking 36 of pedestrian 30. These markings 34, 36 direct 
the attention of the driver of motor vehicle 10 to these objects 28, 30. The image processing 

30 algorithms are designed so that the marking 34, 36 of objects 28, 30 takes place only if 

objects 28, 30 are situated in the area of the course of the roadway of motor vehicle 10, and if 
a dangerous situation is present. In this exemplary embodiment, a dangerous situation is 
recognized because pedestrian 30 is crossing roadway 32 behind oncoming motor vehicle 28, 
and there is the danger of a collision of the driver's vehicle 10 with pedestrian 30. 
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In the two exemplary embodiments according to Figures 1 and 2, image sensors are used that 
have a high resolution. Usable semiconductor image recording chips have resolutions that are 
sufficient for a satisfactory image representation and that enable object recognition at 
distances up to approximately 70 meters. A recognition of objects located further than 70 
meters from the image sensor requires higher-resolution image sensors (imagers) having 
standard resolutions with 1024x768 pixels or 1280x1024 pixels. With standard resolutions of 
1024x768 pixels, an object recognition up to approximately 110 meters is possible, while 
with a standard resolution of 1280x1024 pixels an object recognition up to approximately 140 
meters can be carried out. In the two exemplary embodiments according to Figures 1 and 2, 
the coating of the camera optics is adapted to the spectral range used. The coating is designed 
so that the optical characteristics in the visible spectral range are not significantly worsened. 
In this way, is possible also to use the image sensor system for other functions in daylight, 
i.e., in the visible spectral range. In addition, the aperture of the optics is adapted to the 
prevailing dark sensitivity. 

Figure 3 shows a block diagram of the first and second exemplary embodiments, 
corresponding to Figures 1 and 2, of the apparatus for improving visibility in a motor vehicle, 
made up of an infrared-sensitive image sensor system 12, a processing unit 62, and at least 
20 one signaling means 66. Infrared-sensitive image sensor system 12 acquires optical signals 
from the surrounding environment of the motor vehicle in the form of image data. Via signal 
line 60, the optical signals are transmitted electrically and/or optically from infrared-sensitive 
image sensor system 12 to processing unit 62. Alternatively, or additionally, transmission by 
radio is possible. Processing unit 62 is made up of module 72, shown in Figure 4, which in 
25 these exemplary embodiments is realized as programs of at least one microprocessor. In this 
exemplary embodiment, processing unit 62 is physically separated from the other 
components 12, 66. Alternatively, it is possible for processing unit 62 to form a unit together 
with image sensor system 12, or for processing unit 62 to be housed in signaling means 66. 
Processing unit 62 calculates, from the optical signals of infrared-sensitive image sensor 
30 system 12, signals for driver information. The calculated signals for driver information are 

electrically and/or optically transmitted to at least one signaling means 66 via a signal line 64. 
Alternatively, or additionally, a transmission by radio is possible. From the signals for driver 
information, signaling means 66 produce the actual driver information, for example in the 
form of an optical and/or an acoustic warning. As signaling means 66, in the first exemplary 
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embodiment a display in the multi-instrument panel is used as a first signaling means 66, and 
a loudspeaker is used as a second signaling means 66, while in the second exemplary 
embodiment a projection device is used. 

5 Figure 4 shows a flow diagram of the first and second exemplary embodiments, 

corresponding to Figures 1 and 2, of the method for improving visibility in a motor vehicle, 
and of the apparatus according to Figure 3, made up of processing module 72. Optical signals 
70 are supplied to processing module 72, which calculates, as output signals, the signals for 
driver information 74. The processing module is made up of two modules that work in 

10 parallel; these are the module for roadway course recognition 76 and the module for object 
recognition 78. The algorithms for lane and/or roadway recognition of the module for 
roadway course recognition 76 and for object recognition of object recognition module 78 are 
joined to form an overall algorithm. These two modules 76, 78 exchange information and 
partial results during the processing. In the module for roadway course recognition 76, from 

15 optical signals 70 objects are determined that define the roadway or the lane. These objects 
are for example guideposts and/or lane marking lines. The course of the roadway is 
calculated using the knowledge of the position of these objects. The objects for calculating 
the course of the roadway are determined by evaluating contrasts in the image. For example, 
in order to determine the lane marking lines, contrasts differences between the lane markings 

20 and the roadway surface are evaluated and are traced by the processing unit in the image 

sequence. Irregularities and/or brief interruptions of the lines are corrected, for example using 
Kalman filtering algorithms. In the module for object recognition 78, objects are likewise 
determined from the optical signals. For the object detection, contrast differences between the 
object and its surroundings are evaluated. In a variant of the described method, a 

25 determination of the distance of the detected objects is carried out using a stereo camera. 

From the size of the outline of the object and its distance, it is possible to determine the size 
of the object. Objects that are recognized include in particular oncoming motor vehicles 
and/or pedestrians and/or bicyclists and/or motorcyclists and/or trucks. In a first variant, 
objects are merely recognized as such, while in a further variant an object classification is 

30 carried out. A comparison between a recognized image pattern, for example the shape and/or 
the size of the object, and an image pattern stored in the processing unit forms the basis for 
the object classification. Subsequently, a calculation of the three-dimensional position of the 
recognized objects in relation to the determined course of the roadway is carried out. Using 
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this overall algorithm, it is possible to carry out an allocation of an object to the course of the 
roadway, in particular to the lane and/or to the roadway. 

The described apparatus and the method are not limited to a single image sensor system. 
5 Rather, in a variant additional image sensor systems are used whose optical signals are 

supplied to the at least one processing unit. Here, the image sensor systems are equipped with 
color image sensors and/or black-and-white image sensors. In an additional variant, at least 
one image sensor system is used that is made up of at least two image sensors that record 
essentially the same scene. In another variant, at least one stereo camera is used. 

10 

In a further variant of the described apparatus and of the method, besides at least one like 
pulse and/or at least one warning symbol and/or at least one image marking and/or an 
acoustic signal, alternatively or additionally at least one segment of an image and/or a haptic 
signal are produced as driver information. These possibilities are used either individually or 
1 5 in arbitrary combinations. 

A further variant of the described apparatus contains, besides the at least one optical signaling 
means and/or the at least one acoustic signaling means, alternatively or additionally at least 
one haptic signaling means. These signaling means are used either individually or in arbitrary 
20 combinations. 

In a further variant of the described apparatus and of the method, the processing unit 
alternatively or additionally takes into account the visibility conditions. The visibility 
conditions are defined here by the view of the driver. In this variant, the inclusion of the 
visibility conditions means that the driver is warned only of objects that he himself cannot 
25 see. This results in a reduction of the quantity of information that has to be processed by the 
driver. 

A further variant provides that, instead of the modified headlight with halogen lamp 
according to one of the preceding exemplary embodiments, an infrared-laser-based headlight 
30 is used as a source of infrared radiation. 

In a further variant of the described apparatus and of the method, more than one processing 
unit is used. In this way, a distribution of the algorithms to a plurality of processing units is 
possible. At the same time, there is a redundancy of the required computing capacity, so that 
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when there is a failure of a processing unit, the apparatus and the method for improving 
visibility continue to remain capable of functioning, because the remaining processing units 
compensate for the failure. 

5 In a further variant of the described apparatus and of the method, the use of at least one 
additional sensor enables an improvement of the visibility in a motor vehicle. Through the 
use of at least one additional sensor and the fusion of the sensor signals with the produced 
optical signals, a more reliable recognition of the course of the roadway and/or of the objects 
is attained. As the at least one additional sensor, at least one radar sensor and/or at least one 
10 ultrasonic sensor and/or at least one LIDAR distance sensor are used. The use of at least one 
additional sensor enables the redundant determination of the position of at least one object 
and/or of the course of the roadway. 
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